An autonomous vehicle (also known as a driverless car, self-driving car, or robotic car) is a vehicle that can navigate without direct human control. An autonomous vehicle may analyze the surrounding environment by using a variety of sensing technologies, such as traditional optical image sensors, LiDARs, radars, GPS sensors, IMUs (inertial measurement units), acoustic sensors, odometers, and other sensors accessible via the vehicle's various interfaces. Advanced control systems interpret sensory information to identify appropriate navigation paths, as well as obstacles and relevant signage. Autonomous cars may be equipped with control systems for analyzing sensory data in order to distinguish between different cars or obstacles on the road. Driverless technology has been developed by Google®, Tesla®, as well as other vehicles manufacturers such as Audi®, BMW®, Nissan®, and the like.
Other companies such as, e.g., Mobileye®, are presently trying to provide solutions for hands-free driving technology. Use of this technology is typically limited to particular driving infrastructures such as, e.g., highways or country roads. The cornerstone of such hands-free driving and autonomous vehicle technologies is the rendering or generation of a 3-dimensional (3D) map of a scene at any given moment during or immediately prior to the vehicle's motion. Such a map attempts to mimic a scene digitally as it would normally be seen by a driver. Similar technologies may be applied not only to automobiles, but also to a wide range of drones, including flying drones, walking drones and otherwise mobile drones.
Each imaging technology offers particular advantages as well as disadvantages. Optical sensors, which may be labelled as passive systems, are economical to produce and use to produce an image that is similar to what the human eye can see. Furthermore, commodity image sensors may provide a high resolution, a high density and a high data rate. For example, a typical image sensor of 2K format may have 2.1 megapixels of resolution and may operates at 60 FPS. An optical sensor may be fitted with a wide-angle phi-theta (fisheye) lens to achieve a wide field of view or with a regular (normal) lens to achieve uniform resolution in a narrower field of view.
Optical sensors have limits regarding the dynamic range that they can adequately capture and may require appropriate lighting conditions to produce useful images. If there lacks sufficient ambient natural light, artificial light must be produced to allow the optical sensor to capture a useable image, which can be expensive, cumbersome, and limiting. Certain atmospheric conditions, such as rain, fog or snow, can greatly reduce visibility and limit the effectiveness of a camera using an optical sensor. Optical sensors may also be blinded by excessive illumination producing saturated reading and so called “flare” optical effects. Further, single image sensors alone provide no information about the distance between the camera and an object, and while stereoscopy, i.e., using two sensors or two images with sensor movement in between each, can provide distance information, it has diminishing accuracy beyond 10 meters.
Challenging imaging conditions can be overcome by using a combination of cameras. For example, in some systems, optical RGB cameras may be used during the daytime while thermal cameras can be used during the limited lighting of nighttime. Additionally, multiple wide-angle cameras can be used to provide 360 degree coverage around the vehicle. However, even the use of various optical cameras can still fall short of a desired image result.
Radar devices address some of the shortcomings of optical sensors. A radar is an active system, as it includes a transmitter configured to produce electromagnetic waves in the microwave domain coupled with an emitting antenna. A receiving antenna is configured to capture return electromagnetic waves reflected off of objects positioned in the path of the emitted signal. A radar, compared to an optical sensor, has low angular resolution, due to having a wide angular beam, and therefore provides low-density coverage of the angular field of view. Furthermore, the radars are sometimes configured to have horizontal plane resolution without vertical plane resolution. Therefore, it is mostly appropriate for detecting large objects, especially vehicles. On the other hand, is has excellent accuracy of distance measurement and may cover distance ranges of several hundreds of meters. The radar may also measure the speed of the objects via a Doppler effect reading. Radars are typically not subject to visible lighting conditions and can be equally effective in light or dark environments. Similar to an optical camera, radars are usually economical, both in production and operation. They can be used in certain environmental conditions, such as fog and snow, which severely limit the effectiveness of an optical sensor. Finally, a radar has a strong dependence on the properties of the reflecting objects that it observes, which can be advantageous or disadvantageous, depending on the particular scenario.
LiDAR is similar to radar, but uses a laser light beam to create a virtual image instead of radio waves. The light beam is sent as a pulse in a specific direction toward a scene and a receiver detects reflections from the light beam pulse, which are then used to produce a three-dimensional virtual image. The distance accuracy of a LiDAR is excellent, and its range is limited only by the available peak power of the source laser. As the wavelengths used for LiDAR are much shorter than radar waves the angular resolution of LiDAR beam is typically an order of magnitude better than that of a radar.
LiDAR data rate, however, is limited, and it typically provides about two orders of magnitude fewer samples than an image sensor. Most commercial LiDAR systems use several LiDAR sensors working in tandem to improve the spatial resolutions. In some commercial sensors, like Velodyne® sensors, the LiDARs are mounted on a rotating shaft to produce 360 degrees coverage with horizontal resolution equivalent to the sampling frequency and vertical resolution of the combined number of the scanning LiDAR sensors used. However, combining multiple individual LiDAR sensors increases the cost and size of the LiDAR systems.
Furthermore, unlike radar, LiDAR imaging is limited in certain lighting conditions, such as when facing a bright light source such as the sun, and is subject to deterioration on low visibility or bad weather conditions similar to an image acquired by a camera. Additionally, the use of LiDAR can present safety issues if exposed to a human eye, unless the power of the laser beams is set accordingly, which influences the LiDAR's performance, especially regarding its effective range. Power source constraints, namely the average power of the laser beams, is an additional factor to account for when using LiDAR to detect far objects due to human eye safety constraints, as the LiDAR may be required to conform with Class-1 laser safety standards.
In addition to LiDARs and radars with fixed scanning characteristics, there exists controllable scanners that use micro-electro-mechanical system (MEMS) for LiDAR scanning and Phased Array for radar scanning. The MEMS solution usually includes a system of mirrors that may rotate sufficiently fast for a LiDAR to scan a large area. The phased array solution includes multiple antennas with programmable phase shift between them, so that the resulting response function is equivalent to a larger and quickly moving antenna. However, these controllable sensors are typically less energy efficient than fixed sensors, resulting in a lower sampling rate.
A further form of sensor that can be used to locate objects within a scene include acoustic devices. When used in water, these devices are known as SONARs. Similar devices can be used a dry atmosphere as well. A sound wave, often ultrasonic, or vibrations above the limits of human hearing, is emitted and its acoustic reflections are then detected, which allows a system to determine the shape and movement of an object.
Acoustic sensors offer certain characteristics that exceed performance of optical LiDAR sensors in certain conditions. For example, optical sensors have limited depth of field (DOF) and LiDARs and radars have limited dynamic range. Moreover, these sensors are often designed to be located in an upper part of a vehicle to increase their effective range. Accordingly, acoustic sensors may be placed on a lower part of the vehicle, such as a bumper, to provide distance information for objects that are not visible by the optical sensors, LiDARs, or radars. Moreover, these acoustic sensors are inexpensive to produce and are not affected by atmospheric conditions. Each acoustic sensor typically provides only one distance measurement, and due to its use of sound waves, which travel much slower than electromagnetic waves, the sampling speed is much lower than other sensors. As such, acoustic sensors are typically used for low speed driving and parking situations.
Additional sensors useful for autonomous vehicles include Global Positioning System (GPS) sensors and orientation and motion sensors. GPS provides spatial positioning with accuracy within meters of a location, using triangulation of transmissions from several satellites. Inertial measurement units (IMU) use a combination of accelerometers, gyroscopes and magnetometers to provide orientation and motion data. The combination of GPS and IMU may be used to provide a vehicle's current geolocation, orientation and motion. This information may be used to define road conditions, to compensate the vehicle's motion and to employ external geolocal data, such as predetermined maps and annotations.
The readings from the multitude of sensors may be further combined into a complex environment map, out of which environment parameters may be calculated. The environment parameters analysis may include detection of free space for driving, road detection, lane detection, road sign detection, pedestrian detection, car detection, and other available road element and obstacle detection.
There are several ways used to calculate the environment map, including separate environment analysis per sensor and cross-validation using pairs of sensors. For example, one may detect vehicles in an optical image and measure the distance to the vehicles using a radar.
An alternative method includes rendering of a time-variant 3D map, typically accomplished by measuring distances to many points in the 3D space of a scene to determine the existence of objects and their respective distances from the vehicle.
The rendered 3D maps may be combined and processed to produce the environment model to assist driving decisions made by an autonomous vehicle. Existing solutions for rendering detailed 3D maps are based on LiDAR systems. Such existing solutions configure the LiDAR system to scan the entire environment. This requires a large number of laser measurements to render a single 3D map. Imperfect scene coverage with LiDAR scan may result in the missing or misdetection of an obstacle and increase the possibility of a traffic accident.
For example, FIG. 1A shows an image 100 of a scene for which a 3D map is generated. Some existing solutions for implementing hands-free and autonomous driving technologies rely on a raster scan LiDAR distance sensor for measurement of the distance to each point 110 in the image 100, as shown in FIG. 1B. Thus, if the system is using a LiDAR, a laser beam illuminates each such point to render a 3D map. An alternative method for identifying various objects and areas within a single image using LiDAR technology includes focusing on separate regions of interests as shown in FIG. 1C, such as vehicles 122, pedestrians, 123, stationary structures 124 and roadways 122. LiDAR solutions that are designed to provide optimal measurement density and scan rates are very complex and expensive, and yet trail optical image sensors in both density and data rate by up to two orders of magnitude. As an example, a robotic car currently made by Google® includes equipment with a LiDAR system that can cost up to $70,000. This LiDAR system includes a 64-beam laser and produces high density 3D maps. Similar LiDARs having only 16 beams currently cost only $7,000 but produce a much lower density 3D map. Using a low-cost LiDAR as is produces a low-density 3D map that lacks the definition to reliably identify small obstacles on the road, and is therefore unfitting for autonomous driving. Currently, LiDAR is regarded as a critical component to any solution for autonomous driving; however, due to the high cost of the hardware for rendering the 3D maps, mass production of autonomous vehicles equipped with such systems are not feasible. Thus, a solution is desired that allows using a low-cost LiDAR in tandem with optical imaging and, optionally, a radar and other sensors to produce economically and technically satisfactory results. It should be noted that only a few points 110 are specifically labeled in FIG. 1 merely for simplicity purposes.
It would therefore be advantageous to provide a solution for generating high density 3D maps based on low density distance sensors and high-density image sensor that would overcome the deficiencies of each sensor and the prior art, including measurement resolution, update frequency, minimal and maximal distance, geolocation, weather conditions handling and redundancy for safety purposes.